k3s CA fix: tn-k3s-ca.sh, keyCertSign root, k3s pin v1.36.5 + cgroup v1, no implicit upgrade - #8

CI found it: on Debian/Ubuntu `info` is the GNU info reader. The library fell
back to its own logging only if `command -v info` found nothing, so a
stand-alone `sh tn-k3s-ca.sh check|remediate` on a machine with the info
package printed "No menu item ..." instead of its messages. The library now
logs with tn_ca_info/tn_ca_warn/tn_ca_fatal (same format as header.sh.tmpl).
legacy-ca.sh (harness) asks for a function with `type` for the same reason.

Every unit test now runs with a failing `info` binary first in PATH; a
regression test covers the stand-alone and the setup path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…reign CA, new serial on OpenSSL 1.1.1

- tn_ca_generate_pki: errexit is ignored in the if/|| context of every caller, so the status of
  the subshell was that of its last command and a failed openssl step looked like success.
  Every step now ends in `|| exit 1`, requests go through files (no `-in /dev/stdin`, which
  chroots may lack), and tn_ca_validate_pki checks the new directory (all files, bundle sizes,
  key/cert pairs, chain to the root) before server/tls is backed up and replaced.
- tn_k3s_ca_setup: on an initialised data dir a foreign CA whose root k3s refuses (no keyCertSign
  in a bundle, or expired) now stops the bootstrap ("k3s must not be upgraded") instead of
  letting the unpinned installer run. A foreign root that only expires soon still continues.
- tn_ca_reissue_root: OpenSSL 1.1.1 ignores -set_serial in `x509 -signkey`, so the new root kept
  the serial of the device certificate. Re-issue through `x509 -x509toreq | x509 -req`.
- tests: failing-openssl wrapper (genrsa/req/ca/ecparam), validate_pki, foreign unusable/
  expired/expires-soon, serial of the re-issued root; k3sverify CheckRoot and compat/assert.sh
  assert serial != original (fails on the old library in debian:bullseye and ubuntu:18.04).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…ses, k3s pin

Data dir: check, remediate and setup resolve the data dir the way k3s does
(explicit DATA_DIR/--data-dir, ExecStart --data-dir/-d of the k3s unit,
config.yaml.d/*.yaml, config.yaml, K3S_DATA_DIR of the unit, default). Verified
against rancher/k3s v1.36.5: a command line flag beats config.yaml.d, which
beats config.yaml. check is never ok when a k3s server unit exists but the
resolved dir has no server/tls/server-ca.crt (no-server-data, exit 4).

Regeneration: k3s is stopped (TN_K3S_STOP_CMD / systemctl) while server/tls is
replaced; the old dir is moved aside, the target asserted absent, the installed
result validated, the old dir restored on any failure, k3s started again.

rotate-ca --force also for an old root with no basicConstraints, CA:FALSE, not
yet valid, or whenever openssl cannot verify the old chain but the new one.

Messages: a restart or finalize failure after a successful rotate says the
datastore holds the new CA and how to recover (exit 5); a completed rotation is
finalized instead of rotated twice; leftover ROTATED staging is removed when the
CA is ok; the newest 3 tls backups are kept; '--phase' without a value is a
usage error; a k3s without rotate-ca (< v1.24.12/v1.25.8/v1.26.3) fails safe
with matching advice. Rollback limits documented.

k3s pin: the server and agent scripts install INSTALL_K3S_VERSION
v1.36.5+k3s1 by default; tnctl bootstrap --k3s-version changes it (stable keeps
the installer's own behaviour, latest follows that channel). The server script
keeps a non-default data dir in config.yaml and writes the manifests there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…re-run, final-review fixes

cgroup v1: Kubernetes >= 1.35 makes the kubelet refuse to start on a cgroup v1
host (failCgroupV1 defaults to true; Jetson L4T r32/r35, Ubuntu 18.04/20.04),
and k3s v1.36.5 does not change that. New library pkg/templates/scripts/
tn-k3s-install.sh (embedded in the server and agent script): on a host where
stat -fc %T /sys/fs/cgroup is not cgroup2fs (or cannot be told: OS builds) and
the k3s that will run is >= v1.35, it writes
/etc/rancher/k3s/config.yaml.d/99-teknoir-cgroupv1.yaml with
`kubelet-arg+: fail-cgroupv1=false` (appends to the kubelet-arg of config.yaml;
verified with rancher/k3s v1.36.5: the user's max-pods is kept) and warns that
cgroup v1 is deprecated. Idempotent, removed on cgroup v2, a fail-cgroupv1 set by
the device itself is respected. TN_K3S_CGROUPV1=true|false forces the answer.

Re-run policy: k3s installed -> its exact release is kept (a different pin is
reported and ignored). tnctl bootstrap --k3s-upgrade (TN_K3S_UPGRADE=true) lets
the pin replace it; a downgrade is always refused, an upgrade over more than one
minor is refused unless --force (TN_K3S_FORCE=true). The server script no longer
replaces an existing config.yaml (keys are added when missing).

tn-k3s-ca.sh: an 'activating' unit counts as running when it is stopped before a
regeneration; setup and remediate stop when the resolved data dir was never
initialised but the k3s unit is active and ready (k3s uses another data dir);
prepare checks every write of the bundle rebuild and verifies each rebuilt bundle
(same certificate count, new root last) before rotate-ca; leftover .tn-tls-old.*
and .tn-tls-new.* dirs (they hold keys) are removed once the CA is ok; stop,
start and restart of k3s are bounded (TN_K3S_SERVICE_TIMEOUT, 120 s) and a
timeout is reported as "the CA was changed but k3s did not restart".

Tests: Go unit tests for both libraries and the rendered scripts; harness suite
`install` (real k3s: flag passed and user kubelet-arg kept on >= v1.35, version
read from the real `k3s --version`), install-lib check in openssl-compat.
Docs: bootstrap.md, troubleshooting.md, harness README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…low-device waits, safer refusals

- An agent installs the k3s release of its server, never a newer kubelet. k3s answers /version with
  401 to anonymous callers (all releases), so the server script now publishes the release it runs as
  server/static/teknoir/k3s-version (served by k3s without authentication; tn_inst_publish_release)
  and the agent reads it (then /version, for a server that allows it). A release asked for
  (--k3s-version / INSTALL_K3S_VERSION) is installed if not newer than the server, refused if newer;
  an unknown server release stops the script unless a release was asked for or TN_K3S_FORCE=true.
- Server script order: release policy + cgroup flag, then the CA step, then the installer, so a k3s
  that a cgroup v1 host made exit has the kubelet flag when the CA step restarts it; rotate-ca is tried
  again while the unit starts.
- TN_K3S_SERVICE_TIMEOUT defaults to 600 s; after a timeout the unit is watched 120 s more.
- A k3s binary or server/db whose version cannot be read is not installed over (TN_K3S_FORCE=true).
- fail-cgroupv1=false from k3s v1.31 (was v1.35), written atomically.
- Refusals say what this run changed (TN_K3S_CA_CHANGED); docs fixed.
- A data dir that differs from config.yaml(.d) is refused before anything is changed.
- Harness: install suite covers v1.30/v1.31 and the agent-follows-server scenario with real k3s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…a dir guard first, harness expects the drop-in from v1.31

The server script rebuilt INSTALL_K3S_EXEC after tn_inst_prepare. The library reads that variable
(a fail-cgroupv1 in it counts as the device's own setting and removes the drop-in), so the order
mattered: build it first, keeping the caller's value, so the library sees what the installer gets.

- server.sh.tmpl: INSTALL_K3S_EXEC is built before tn_ca_resolve_datadir / tn_inst_prepare
- new tn_ca_require_consistent_datadir (library): called right after tn_ca_resolve_datadir so a data dir
  that disagrees with config.yaml refuses before tn_inst_prepare changes anything
- test/k3s-ca/compat/assert.sh: the drop-in is expected for an installed k3s 1.34 (written from v1.31),
  and none for 1.30
- stale strings: agent header and troubleshooting say /static/teknoir/k3s-version first; cgroup flag
  from v1.31; fleet note: a new agent stops until the server script is re-run (it also repairs a
  legacy CA) or INSTALL_K3S_VERSION is given
- tests: the caller's fail-cgroupv1 reaches the installer, order, data dir guard first

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t the flag in the unit

tn_inst_cgroup removed 99-teknoir-cgroupv1.yaml at tn_inst_prepare time whenever the
device's own fail-cgroupv1 was found, also when it was only in INSTALL_K3S_EXEC. That
reaches the unit only when the installer has run, so a k3s restart in the gap (the CA
step restarts k3s, or the device reboots after the script stopped) left a cgroup v1
k3s >= 1.35 without the flag.

- The drop-in is removed early only when the device's setting is in a persistent config
  file that k3s reads (config.yaml, config.yaml.d/*.yaml); files win over INSTALL_K3S_EXEC.
  With only INSTALL_K3S_EXEC the drop-in stays or is written (same value twice is harmless,
  the command line wins).
- Audit: writing the flag for an installed k3s older than v1.31 (about to be upgraded)
  would stop its kubelet on a restart before the installer. That write now waits for the
  new tn_inst_finish, called right after the installer in the server and agent scripts;
  an existing drop-in is removed at once in that case. The other removals (host on cgroup
  v2, release older than v1.31) are safe early.
- Library version 3. Tests: go, dash, and the openssl-compat image check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…its for the installer

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… its first start

When the drop-in has to wait for the installer (installed k3s older than v1.31) and the
release the installer will start is a channel or v1.35 or newer, --kubelet-arg=fail-cgroupv1=false
is appended to INSTALL_K3S_EXEC (once), so the new unit carries it from its first start and the
installer's systemctl restart does not fail before tn_inst_finish. The old kubelet never reads it.

Only an uncommented fail-cgroupv1=false (config.yaml, config.yaml.d/*.yaml and *.yml,
INSTALL_K3S_EXEC) counts as the device setting it itself; a comment or =true does not, and
does not remove the drop-in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>